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ABSTRACT 



The program development process for substance abusers is 
very challenging. Planning and evaluation in substance abuse prevention and 
treatment is continuously affected by change. It requires extraordinary 
flexibility, adaptability, and creativity to insure that programs are 
designed to effectively meet the multiple needs of the clientele. Counselors 
must not only be committed to staying abreast of tht current best practice^j 
in the broad field of human services, but they must also critically examine 
and evaluate those practices in an ongoing quest to find more effective 
approaches to critical problems. This is especially true in the area of 
substance abuse where treatment outcomes have been consistently poor. This 
chapter presents a six step approach to planning and evaluation. To 
facilitate the recall of the six steps, the RAPPER Model is offered. It 
requires the use of each letter to recall the key first words which describe 
the steps in the process: review the literature; assess the need; plan goals, 
objectives, and strategies to achieve them; practice those strategies which 
best address the needs; evaluate the effectiveness of the chosen strategies; 
and repeat effective strategies. As a model that emphasizes a commitment to 
continuous improvement based upon ongoing program planning and evaluation, it 
should be viewed dynamically, like a wheel that is in constant motion. Also 
included is a facilitator's manual which is a practical guide for teaching 
content through guided experiential projects. Objectives are listed for each 
lesson. Eleven student activities are provided. (MKA) 
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Program Planning and Evaluation 

Robert Rapp 

Introduction 

In substance abuse treatment and prevention programs, success depends as much on competent program 
planning and evaluation as it does on competent service delivery. Substance abuse programs and the needs that 
they address are dynamic; there is only one constant. That is constant change which dictates that program 
planning and evaluation be an ongoing process, if the programs which are developed are expected to be effective 
and remain effective. The initial goal of this process is to determine what is believed to be the most effective 
way to address a need. Then, the more challenging aspect of the process is to implement a system of evaluation 
which will continuously seek to improve the program through the ongoing discovery of even more effective 
ways to deliver the needed services. 



U.S. DEPARTMENT OF EDUCATION 
Office of Educational Research and Improvement 

EDUCATIONAL RESOURCES INFORMATION 
CENTER (ERIC) 

O This document has been reproduced as 
received from the person or organization 
originating it. 

O Minor changes have been made to 
improve reproduction quality. 



Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



The Process 

The program development process encompasses six steps: 

1. Recognizing the Problem and Reviewing the Literature. 

2. Completing a Comprehensive Needs Assessment. 

3. Formulating Goals and Objectives. 

4. Determining the Methodology. 

5. Implementing the Program. 

6. Evaluating the Program. 

Those who are familiar with grant writing can readily see that this process includes the same components 
which are included in all well developed proposals. In fact, for new programs which require obtaining funding 
for implementation, this process and the development of a grant proposal are essentially linked together. The 
written grant proposal is generally the initial product of the program development process. 

Step One: Recognizing the Problem and Reviewing the Literature. 

Recognition of the severity of a problem such as substance abuse requires becoming as well informed as 
possible, which usually begins with an in-depth review of the literature. Comprehensive knowledge related to 
causes, treatment, and the impact of the problem is essential in the program planning and evaluation process. 
This task helps us to determine the extent of the problem and what is being done programmatically about the 
problem. It should also help us determine what not to do in our own program. In substance abuse, there is a well 
documented history of ineffective treatment. The rate of recidivism among substance abusers is extremely high. 
Typically, three out of four substance abusers will relapse or return to using alcohol or drugs after treatment. 

Therefore, a careful review of the program literature can reveal as much about ineffective treatment as it 
does about effective treatment. The challenge is to absorb all this information and to use it in the program 
planning and evaluation process to develop the most effective program possible. 

Step Two: Completing a Comprehensive Needs Assessment. 

This step is the key to the entire planning process. The success or failure of a program can be directly 
attributed to the completeness and accuracy of the needs assessment. To insure that a needs assessment is as 
accurate as possible, it must be comprehensive. A comprehensive needs assessment includes several procedures: 
a clear identification of the target population, a thorough review of the social indicators directly pertinent to the 
target population, a survey of the community of which the target population is a part, a survey of other agencies 
who are either serving or planning to serve the target population, and extensive communication with key informants 
who have special knowledge or relationships with the target population. 

Identifying the Local Target Population and Reviewing Social Indicators 

A clear identification of the local target population and a thorough review of the directly pertinent social 
indicators go hand in hand. Without careful scrutiny during this procedure, the target population and the actual 
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persons receiving services may not be one and the same. 

Related to that, a few years ago, the United Way requested proposals to provide delinquency prevention 
services to at-risk children and youth in the city of Tucson. Two well funded organizations whose services were 
geared toward middle-and upper-middle-class children in well-to-do neighborhoods were seeking the limited 
amount of funds which were available. The other agencies who were competing for the delinquency prevention 
funds, all provided services in areas of the community which had high poverty rates, high crime rates, high rates 
of substance abuse, high drop-out rates, and ongoing gang activity. 

The two well-funded organizations, who had several supporters among the United Way Allocations 
Committee, justified their requests by arguing that all children are at-risk of becoming juvenile delinquents. This 
argument, of course, has some merit, but the degree of risk should be considered. When social indicators pertinent 
to the target population were studied, the picture became very different. Social indicators are most often presented 
as statistics related to social need. The statistics representative of the social needs of the children targeted by the 
other agencies clearly demonstrated where the funding for services should be directed. Fortunately, the information 
gleaned from the review of the social indicators directly pertinent to the target population was so compelling that 
the funding was properly directed. Unfortunately, that is not always the case. Without careful attention to this 
part of the process, funding may be misdirected and the needs of the target population may never actually be 
addressed. 



Surveying the Community 

For a comprehensive needs assessment to be valid, the community, of which the target population is a part, 
should always be surveyed. The survey can include several different procedures. It can include questionnaires 
which may be distributed to community members through schools, businesses, the media, and other organizations. 
It can include open forums and meetings where the community is invited to provide input. It could include 
telephone interviews and personal contacts. The survey should always include substantial participation from 
members of the target population. At times, it could represent a sample of the target population. At other times, 
it may include the entire population, such as when a questionnaire is administered to an entire school of students 
or other representative groups. The key is that the information gathered from the target population itself and its 
immediate community is usually the most reliable and the most valuable. How often are services offered to a 
community where there are a limited number of persons who utilize them? When this occurs, how often do 
program staff complain that the community isn’t using the services because “they don’t know what’s good for 
them?” That conclusion is especially presumptuous when the target population and its community has been 
excluded from the planning process. 



Surveying Local Agencies 

A survey of other organizations which are providing services or are planning to provide services to the 
target population is also mandatory. Lewis and Lewis (1991) depict the folly of two organizations who have 
targeted the same population for mental health services. Even though one organization has worked closely with 
the community, and the other has not, neither is able to effectively address the community’s mental health needs 
and potential funding is lost because they have not collaborated and will duplicate services. (Case 2-1. Community 
Action and Mental Health, pp 71-72). There certainly is no need to duplicate services when there is a plethora of 
unmet human services needs. The challenge lies in addressing them collaboratively, comprehensively, and 
effectively, so that services provided are not just band-aids to temporarily stop the bleeding, but which never 
lead to permanent changes for the better. 

When every agency knows what the other local agencies are doing or are planning to do, then staff are able 
to network with one another and develop formal linkages with one another which can comprehensively address 
needs, so that the “usual” gaps in services are eliminated. 

Communicating With Key Informants 

Key informants are individuals or members of organizations who are directly in touch with the target 
population and its community. They usually are knowledgeable about community needs and how they should be 
addressed. They are often recognized as local leaders and are able to exert a considerable amount of influence 
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over the reception of human services which are offered in their communities. They are valuable allies in both the 
program planning and the program evaluation process. 

Key informants can play an important part in conducting a comprehensive needs assessment. They can 
help organize and participate in the open forums and meetings. They can participate in the design and 
implementation of the surveys of the target population, the community and local agencies. They can identify 
other individuals who should be included. In short, they are essential to the program planning process. To ignore 
their input and participation in the process will likely guarantee that a project will be ineffective in addressing 
local needs. 

In reflecting upon this, I am reminded of a situation in which an organization serving homeless persons 
wrote a proposal and obtained funds from the City Council to lease a vacant facility on West Congress Street in 
Tucson, Arizona, through which they intended to provide local services. During the planning process, they 
ignored the steps necessary to conduct a local comprehensive needs assessment and based all need upon statistics 
related to the entire homeless population of Tucson. A greater flaw was the exclusion of key informants. In this 
case, the Tucson West Neighborhood Association members represented the most influential group of key 
informants in the community. To make a long story short, the Tucson West Neighborhood Association opposed 
the services, the City Council withdrew the funds and the organization never succeeded in offering any services 
in that community. 

Interestingly, an organization which the author directed, planned and built a human services center in the 
same community less than two years later which included a 12-unit apartment complex to serve homeless 
families with children. This time, the Tucson West Neighborhood Association did not oppose the program. 
Instead, they committed their share of Community Development Block Grant (CDBG) funds to help build the 
facilities. The difference was that members of the Tucson West Neighborhood Association were included from 
the beginning of the planning throughout the process because they were recognized as key informants who were 
essential to the success of the program. 

Step Three: Formulating Goals and Objectives 

The most common difficulties of this section are related to a general lack of understanding of the difference 
between goals and objectives. Often, these two terms are used interchangeably. When this occurs the results are 
usually unclear or “fuzzy” goals and objectives which are difficult or impossible to measure. Goals are broad 
“mission” statements which are not measurable without specific objectives. For example, a goal statement 
related to our mission could be “the rate of substance abuse among teenagers in Silver City, New Mexico will be 
significantly reduced.” This is a perfectly reasonable and worthy goal for a substance abuse prevention and 
intervention program. The problem is how does one clearly measure a significant reduction in the rate substance 
abuse so that we know our goal has been achieved? 

Specific objectives related to this goal enable us to ascertain that the goal is met. Three specific objectives 
related to this goal could be: 

1. The arrest rate for DWI by teenagers in Silver City, New Mexico between January I, 1999 and 
December 31, 1999 will be reduced by 25%. 

2. The number of teenagers in Silver City, New Mexico who are referred for detoxification between 
January I, 1999 and December 31,1999 will be reduced by 25%. 

3. The number of teenagers in Silver City, New Mexico who are referred to Juvenile Probation and 
Parole Office for substance abuse related offenses will be reduced by 25% during the period January 
I, 1999 and December 31, 1999. 

All of these specific objectives are easily measurable and a 25% reduction during a one year period would 
reflect a significant reduction in the rate of substance abuse. 

Specific objectives clearly address the target population which is identified in the goal statement. In addition, 
they are always time oriented and measurable. When specific objectives are met, the extent to which the goal is 
achieved and its impact are readily understood. Also, it should be emphasized here that as soon as goals and 
objectives are formulated, the evaluation process should be initiated, in that, it should be simultaneously 
determined how the objectives will be measured and how their achievement will be documented. In the above 
examples, data should be collected from the proper authorities at the beginning of the project period and again 
at the end so that the actual reduction can be measured and documented. 

Step Four: Determining the Methodology 

This determination actually begins with the review of the literature. However, as was stated in Step One, 
that review helps us determine what not to do as much as it helps us to determine what to do in our treatment 




4 



methodology. The key to determining effective methodology is most often related to our creative efforts in 
addressing the problem. More of the same old thing should not be expected to produce better results! Instead, we 
must focus upon what we can do differently to achieve better results. A guide to stimulate our creativity in order 
to meet the challenge to find a more effective approach to the problem can be very valuable in this step. 

Wubbolding (1988,1991) developed the WDEP model to help remember the procedures of Reality Therapy: 

WDEP MODEL 

W = What do you want? 

D = What are you doing to get what you want? 

E = Evaluate it! Is it working? 

What else could you do? 

P = Develop a plan to get what you want! 

When applied to the program planning and evaluation process, Wubbolding’s model can also be very 
helpful. Used in the plural, i.e., “What do we want? What are we doing to get what we want?”, etc., the W.D.E.P. 
model can be effectively used to stir our collective creative juices to respond to the challenge to find a better way. 

Any information that can be useful in effective prevention and intervention programs should be strongly 
considered for inclusion within the design of the methodology. For example, some information that this author 
has found to be very valuable related to prevention and intervention programs for children and adolescents 
includes the following. 

Erik Erikson (1963, 1982) taught us the importance of mastering psychosocial challenges during the various 
stages of growth and development. From the stages of Infancy to Adolescence, Erikson has demonstrated the 
importance of developing a sense of trust, developing self reliance and initiative, developing a sense of competence 
and developing a healthy identity. If these core psychosocial challenges are not adequately met, children will 
move through the various stages of development with feelings of mistrust, self doubt, shame, guilt, inferiority 
and will be mired in role confusion during adolescence. Not coincidentally, adolescence is the time when children 
are most vulnerable to substance abuse and other self destructive behaviors. 

William Glasser (1992, 1998) teaches us that all behavior is purposeful. It represents an attempt to meet 
one or more of our basic needs for survival, love and belonging, power (recognition or self-worth), freedom and 
fun. The foundation of his teaching is that our ability to effectively meet our needs within the structure of our 
society is determined internally through the choices that we make and not as a behavioral response to environmental 
stimuli. This approach promotes the development of a strong internal locus of control which empowers individuals 
to take effective control of their lives. 

In Preparing Tomorrow's Teachers In Substance Abuse Prevention (1991), Moore states that “without the 
skills of genuiness and respect, teachers are not likely to be successful in building strong relationships with 
students . . . .Teachers who are respectful allow their students to develop their own solutions believing that they 
have the capacity to be self-responsible.” 

In the same work, Kline (1991) states. 

Teachers can support prevention of substance abuse and other health-compromising behaviors by gaining 
student trust and respect, and helping students to: 

• Recognize that they can confront feelings of insecurity, rejection, and failure. 

• Overcome obstacles by creative planning that increases their autonomy. 

• Channel their energy into more productive and self-actualizing ways to foster self-respect. 

Amembal (1991) in the same work, states. In most schools, the peer group provides the primary means for 

satisfying the affiliation needs of students. Because the norms of peer group affiliation may be in conflict with 
educational and prevention activities, it is important strategically for educators to learn to rechannel this powerful 
influence into constructive avenues. . . 

Another construct which I believe is essential in every program for children and adolescents, I call ‘The 
Breakfast Club Effect.” ‘The Breakfast Club” is a movie which stars Molly Ringwald as a high school “princess”, 
Jud Nelson as a “criminal”, Emilio Estevez as an “athlete”. Ally Sheedy as a “basket case”, and Anthony Michael 
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Hall as a “brain”. This very different group of high school students had all violated school rules in some way and 
had to come to Saturday detention as their punishment. As their day together begins, they spend most of their 
energy trying to hurt each other, in an apparent attempt to validate their own status and their own particular 
clique. As the day wears on, they become united against the pompous vice principal in charge of their detention. 
Through this common purpose they finally begin to honestly communicate with each other and they discover 
each others’ vulnerability and uniqueness. Before the day ends, the criminal and the princess become a couple, 
the athlete and the basket case become a couple and the brain becomes everyone’s best friend. 

The vice principal demanded that they write an essay on the topic “Who do you think you are?” Before 
they could leave detention, the “brain” who completed the assignment on behalf of the entire group wrote: 
Dear Mr. Bernard; 

We accept the fact that we had to sacrifice a whole Saturday in detention for whatever it was we did 
wrong. What we did was wrong. But we think you’re crazy to make us write an essay telling you 
who we think we are. What do you care? You see us as you want to see us, in the simplest terms, 
with the most convenient definitions. You see us as a brain, an athlete, a basket case, a princess, and 
a criminal. Correct? That’s the way we saw each other at seven o’clock this morning. We were 
brainwashed. But what we found out is that each one of us is a brain and an athlete and a basket 
case, a princess, and a criminal. Does that answer your question? 

Sincerely yours. 

The Breakfast Club 

Since discovering this movie, this author has embraced “The Breakfast Club Effect” as a valuable 
programmatic construct to address the needs of children and adolescents. When youngsters who appear to be 
greatly different from one another are given the same status and are put into positions and situations where they 
work closely together, communicate with each other, and depend upon one another to accomplish various tasks, 
they are able to break through the barriers between them. They learn that they are really very much alike with 
the same needs, fears, and dreams and they often become fast friends. At the very least, they learn to understand 
and accept each other. 

All of the above is very valuable information which is interrelated. It mandates that human services 
professionals create a warm, supportive, and trusting environment in which students can effectively meet their 
psychological needs for love and belonging, power (recognition and self worth), freedom and fun in every 
program. It also mandates that the program emphasize activities in which students can develop self reliance or 
autonomy, initiative, and competence through which their energy will be rechanneled into “more productive 
and self actualizing ways to foster self respect.” This will enable adolescents to move beyond the characteristic 
role confusion, toward a positive and healthy identity, with an understanding and acceptance of those who 
appear to be different. 

Information from many other authorities on human behavior can also be very helpful and may enable us to 
develop more effective programs. What will actually determine program effectiveness is our ability to creatively 
integrate useful information into the program methodology design, and then put it into practice. 

Step Five: Implementing the Program 

Diligent completion of the first four steps can enable us to find the support necessary to implement our 
program. If funding is required, then a budget must be developed. For persons who are not experienced in the 
development and management of programs, the concept of developing a budget can be intimidating. It should 
not be. “The budget itself is simply a projection of operational plans, usually for a one-year time span, with the 
plans stated in terms of the allocation of dollars for varying functions or activities” (Lewis and Lewis, 1991). It 
is a tool which enables us to translate our goals and objectives into actual practice. Some basic calculation skills 
are needed to develop an accurate budget, but one need not be an accountant. 

Budgeting is clearly tied to the planning and evaluation process. Therefore, the most capable persons to 
develop the budget are the human services professionals who are most capable of planning, implementing, and 
evaluating human service programs. Relegating the formulation of a budget to bookkeepers or accountants, 
who are not human service professionals, will limit the agency’s ability to effectively meet its goals and objectives. 
Feldman (1973) notes that a budget can rigidly control expenditures and limit the discretion of program 
administrators. Hodges (1982 ) emphasizes that “Budgets are part of the planning phase. As a result, budgeting 
incorporates not Just the monies to be expended but all of the assumptions related to need, objectives, and ability 
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to deliver services that are found in the planning phase” . 

Therefore, human service professionals should not shy away from the budgeting process. Instead, they 
should embrace it as an integral part of the planning process. Accurate costs simply must be calculated for the 
personnel and services needed to operationalize your program plans. The most typical cost categories are 
Personnel, Employee Related Expenses, Professional and Outside Services, Travel, Space, Equipment, Materials 
and Supplies. Anything else generally is assigned to the category of Other Operating Costs. The ability to 
project and accurately calculate cost in these line item areas is all that is needed to produce a budget which 
transforms program goals and objectives into actual practice. 

Implementing effective prevention and intervention programs for children and adolescents is not always a 
costly venture. For example, following the above planning process and using the knowledge gleaned from the 
works cited in the methodology section enabled this author and a high school counselor to implement a highly 
effective prevention and intervention program at Cobre High School in Bayard, New Mexico. The annual costs 
of operating the program compared to its benefits were very small. At most, the costs were $5,000.00 per year. 

Initially, the program started with less than 20 adolescents who were primarily honor students. It eventually 
grew to approximately 80 students who represented a cross section of the Cobre High School student body. This 
number represented about 12% of the entire student population and was the largest single group of students 
involved in an extracurricular activity at the school, including the individual athletic teams. The program was 
designed to include honor students, athletes, cheerleaders, ropers, nerds, and students who had abused drugs 
and alcohol as well as students who had been gang affiliated. The program originally focused upon substance 
abuse prevention and intervention, but expanded its focus to include gang prevention and intervention when 
that became a problem at school and in the community. 

The program’s success was largely due to the inclusion of diverse groups of students who were representative 
of the entire student body and the program’s creative design which emphasized the concepts discussed in the 
section on methodology. Trust was developed between the adult sponsors and the diverse participants through 
specifically designed activities. Program ownership was offered to the students through the opportunity to 
develop and control the program, in contrast to programs that are generally imposed upon the students by their 
adult sponsors. 

The students created dramatizations which vividly portrayed the real dilemmas which affect and seriously 
impact children and adolescents today. These dramatizations often realistically portrayed the tragedies which 
often result from substance abuse and gang violence. They also developed several dramatizations which depicted 
the influence of parents’ behavior upon the behavior of their children. These dramatizations were usually 
performed before younger student audiences to help prevent them from choosing to use drugs or to join gangs. 
The presentations were followed by a discussion between the older and younger students on the situations 
portrayed through the dramatizations. 

The high school students developed confidence and competence through their presentations. Their 
commitment to help younger students also helped the high school students to make better choices related to 
their own experiences with drugs and gangs. Many of these teens also underwent extensive training to become 
peer counselors at the High School, the Middle School, and two elementary schools. Eventually, hundreds of 
students benefitted from this inexpensive, yet creative and effective approach to prevention and intervention. 

Ultimately, a group of these students were invited to make a presentation before several hundred persons 
from 14 different countries at The International Convention of The William Glasser Institute held in Albuquerque, 
New Mexico, in July, 1996. Subsequently, two professional videos were produced featuring many of these 
students in their prevention activities (Rapp, 1997). 

This program was highly successful because it offered the students opportunities to meet their psychosocial 
needs for love and belonging, recognition, freedom, and fun; and opportunities to master psychosocial 
developmental challenges in positive ways. The inclusion of a cross section of the student body and emphasis of 
need fulfilling activities within the program’s design were essential to its success. In addition, the fact that the 
program was owned and operated by the teens magnified its influence among their peers. Tobler (1986) conducted 
research which concluded that peer programs are more effective than other program modalities in reducing 
substance abuse among teens. Tobler found that peer programs were very effective for the average school based 
adolescent program. However, for at-risk adolescents, including drug abusers and juvenile delinquents, peer 
programs were shown to be highly successful. 

Related to that, a recent report of the Carnegie Council on Adolescent Development (1996) concluded 
that, “all students must find ways to earn respect, establish a sense of belonging in a valued group and build a 
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sense of personal worth based upon the mastery of useful skills, including social skills.” 

Program implementation can be exciting and rewarding when the program planning process is followed 
carefully and is valued by the planners. It insures that effective approaches to prevention and treatment will be 
implemented and validated through the program evaluation process. 

Step Six: Evaluating the Program 

An appropriate preface to this section is the belief that we should be on a constant quest to find a better 
way to do things, and as soon as we find it, we should start looking for an even better way. This statement 
emphasizes the importance of ongoing program evaluation. Program evaluation is not very well understood; 
many service providers view it negatively. Many see it as just another level of bureaucratic interference that 
requires more paperwork and less time for direct services to clients. For those who view it that way, evaluation 
procedures will be imposed upon the program externally from funding sources and other stakeholders, and its 
value will be diminished. 

However, when program evaluation is internalized within the program development process, it is clearly 
one of the most important components related to program effectiveness .Without it, the effectiveness and 
impact of a program cannot really be determined. Important decisions related to the program will be based 
upon external perceptions, rather than facts which can establish program effectiveness. 

Glasser (1994), in his book. The Control Theory Manager, expounds upon the importance of managers 
promoting on-going internal self evaluation as a basic condition that will improve quality in the workplace. He 
points out that external evaluation by the “boss” will never produce the quality work which can be produced 
through self evaluation and continued improvement. He states that, “the traditional ‘an inspector inspects, 
passes or rejects the work’ is static and does not involve the worker. It will lead to passable, even good work, 
but very little quality. Treated this way, workers will rarely, if ever, do what they are capable of doing.” 

Externally imposed program evaluation can be viewed similarly. It will probably rarely, if ever, produce 
quality in the delivery of human services. Instead, program staff who do value evaluation will design internal 
ongoing (self evaluation) procedures which reflect a commitment to continual improvement. This is what will 
enable quality programs to rise above those who are content to do the same thing over and over with very little 
regard for effectiveness. 

Tripodi, Fellin, and Epstein (1978) define the three basic objectives of program evaluation as: 

I .To provide descriptive information about the type and quantity of program activities (program effort). 

2. To provide information about the achievement of the goals (program effectiveness). 

3. To provide information about program effectiveness relative to effort (program efficiency). 

Gathering and analyzing data about program effort, effectiveness, and efficiency provides information to 

make important management decisions related to the program. It influences decisions about the ongoing need, 
modifications affecting the size and scope of the program, its impact upon the problem, and substantiation of its 
value to the schoofcommunity, or both. 

Designing an evaluation plan which is clearly tied to the program’s goals and objectives should provide 
adequate measures of effort, effectiveness, and efficiency to satisfy all of the stakeholders in the program. The 
most common stakeholders in prevention and intervention programs which target children and adolescents are 
parents, teachers, counselors, school administrators, funding sources, and the community. 

As was mentioned in Step Three, the program evaluation process actually begins concurrent with the 
formulation of goals and objectives. As these are formulated, as part of the evaluation plan, there must be 
means identified to collect data which enables us to accurately assess effort, effectiveness, and efficiency. This 
can only be accomplished through a Process Evaluation and an Outcome Evaluation which measures and 
documents the achievement, lack of achievement of goals and objectives, or both. 

The procedures used to document effort are often referred to as the Process Evaluation. The first concern 
of the Process Evaluation is to determine whether the target population has actually received the services 
according to the program plan. An example of this aspect of the Process Evaluation can be illustrated through 
the following specific objective: 

Five hundred at-risk children from disadvantaged neighborhoods in Tucson will be enrolled in 
delinquency prevention activities between January I, 1999 and December 31, 1999. 

Apropos to this evaluation concern is the situation cited earlier where a well-heeled organization sought 
funding from The United Way to provide delinquency prevention services to at-risk children using the reasoning 
that all children were at-risk. Should that organization have been successful in obtaining the funds, whatever 
they achieved would have little to no value in addressing the needs of the target population, because at-risk 
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children, to any significant degree, would not have actually received the services. 

The other concern of the Process Evaluation is to determine the number and types of service activities 
which are delivered to the target population. This determination is linked to the planned treatment methodology. 
An example can be related to the following specific objective: 

During the period January 1,1999 through December 31, 1999 each of the program participants 
will receive at least fifty hours of individual counseling focused upon anger management 
techniques. 

If these services and others which are indicators of the program’s effort are not actually accomplished, 
then the process is flawed. If the target population has not actually been served to the degree planned, then the 
process is flawed. A flawed process invalidates any outcomes which have been achieved. 

The procedures used to document effectiveness are often referred to as the Outcome Evaluation. The 
specific objectives related to reducing substance abuse which were used as examples in Step Three are all 
expected to yield measures of effectiveness and are components of the Outcome Evaluation. If data is collected 
through the evaluation process which documents a 25% reduction in the arrest rate for DWI, in referrals for 
detoxification, and in substance abused related referrals to the Juvenile Probation and Parole Office in Silver 
City, New Mexico between January 1, 1999 and December 31,1999, then the program’s effectiveness will have 
been documented through this component of the Outcome Evaluation. 

Another very important component of the Outcome Evaluation is the development and implementation of 
well-planned follow-up procedures. In order to document program effectiveness over a period of time, follow- 
up procedures enable us to track clients’ progress beyond the immediate analysis pre-treatment/post-treatment 
data. Caddy (1980) emphasizes the importance of establishing follow-up procedures beginning with the initial 
intake interview, stressing it throughout treatment, and obtaining the client’s commitment to participate in the 
process subsequent to treatment. This approach improves clients’ cooperation and enables them to understand 
that the program’s interest in their welfare extends well beyond the formal treatment process. Frequent contact 
with clients after formal treatment, strengthens the client/therapist relationship which is need satisfying and 
serves as a form of ongoing treatment that helps prevent problems from occurring and provides immediate 
intervention or both. 

Effective follow-up procedures can be enhanced through the use of well designed instruments which 
incorporate core indexes of wellness which should be used in all substance abuse treatment/evaluation studies. 
Emick and Hansen (1983) suggest that the core indexes should include the following criteria: treatment 
completion, recidivism, mortality, treatment use, physical health, drinking behavior, other substance use, legal 
problems, vocational functioning, family/social functioning, and emotional functioning. 

The following instruments incorporate useful core indexes which have been reproduced in a handbook by 
Letteri, Nelson, & Sayers (1985) which was developed for The National Institute on Alcohol Abuse and 
Alcoholism: 

Addiction Severity Index (McLellan, Luborsky, Woody & O’Brien, 1980). The ASI assesses medical 
status, employment, drug use, alcohol use, legal status, family/social relationships, and psychological status. It 
is well-tested and has been found useful for ongoing follow-up. 

ATC Client Progress and Followup Form (National Institute on Alcohol Abuse and Alcoholism, 1979). 
This instrument was developed to follow-up clients six months after their treatment at NIAAA programs. It 
addresses marital status, employment, financial support, drinking behaviors, motor vehicle records, 
institutionalization, and client self perceptions. 

ATC Followup Study Questionnaire (Ruggles, Armor, Polich, Mothershead, & Stephen, 1975). This 
questionnaire was developed for use in 18 months follow-up interviews with clients who were treated in NIAAA 
programs. It surveys family status, employment, alcohol consumption, treatment history, legal problems, and 
perceptions of drinking problems. 

Behavior Rating Scale-Social, Employment, Economic, Legal, Drinking (Brandsma, Maultsby, &Welsh, 
1980). This instrument is used before treatment and in follow-up. The authors used this 64-item scale as a pre- 
treatment/post-treatment tool to follow-up problem drinkers who had completed an outpatient treatment program. 

Client Follow-up Interview (Kelso & Fillmore, 1984). This instrument is used at intake, discharge, and 
follow-up. It addresses psychological functioning, alcohol consumption, drug use, physical health, personality, 
treatment, social relationships, employment, legal issues, life events, attitudes, and coping skills. 

Health and Daily Living Form ( Moos, Cronkite, Billings, & Finney, 1984). This instrument was developed 
for a longitudinal follow-up study. It can be self administered or used by an interviewer. The HDL assesses 
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social functioning, health related functioning, resources, family and home environment, children’s health and 
functioning, life-change events, and coping skills. 

The data gathered through the Process Evaluation and the Outcome Evaluation is used to determine program 
efficiency. Efficiency Evaluation procedures are intended to make judgments connecting costs to outcomes. 
They represent a ratio between effort and effectiveness (process and outcome) which is deemed acceptable to 
address the needs of the target population. The critical question is, “Can the program achieve the same or better 
results either by reducing the effort or by providing less costly methodology?” To be very blunt, the programs 
success must be financially feasible. Therefore, the ratio between successful treatment and cost is a major 
consideration in documenting the value of a program. 

A program designed to prevent or treat substance abuse must be proven to be cost effective. If it is not 
determined to be cost effective, it either will not be initiated, or it will not be continued after it is evaluated. For 
example, any program designed to operate on the basis of a one-to- five staff/client ratio would be expected to 
produce a larger percentage of successful treatment than a program designed to operate on a one-to-twenty-five 
staff/client ratio. However, even if the first program produced an 80% success rate compared to a 40% success 
rate for the second, it would not necessarily be judged to be more valuable. 

From an effort/effectiveness ratio (cost/outcome), the first program would probably be judged inefficient 
and its value would be diminished. To clarify that from the standpoint of dollars invested, assume that each staff 
costs $50,000 in terms of salary, benefits, space, travel, etc. In the first program, the cost of successful treatment 
would be $12,500 per client, for the second, the cost of successful treatment would be $5,000 per client. Related 
to that, the second program would succeed with ten clients (40% of 25) per each staff; whereas, the first would 
succeed with only four clients (80% of 5) per each staff. Therefore, the 80% success rate of the first program 
would not be highly valued. 

On the other hand, a program with a one-to-twenty staff/client ratio which has a 60% success rate should 
be judged more effective and more efficient than either of the other two, because its cost of successful treatment 
per client would be $4167. It would succeed with 12 clients (60% of 20) per each staff member. Therefore, of the 
three programs, the third more satisfactorily responds to the critical question related to program efficiency. 

A well designed program evaluation plan which documents effort, effectiveness and efficiency through the 
achievement of goals and objectives will not only satisfy the demand for accountability, but will also enable a 
program to broaden its base of support. This is especially critical during the current period of shrinking resources 
in which programs are required to compete for needed funds. Programs which intimately embrace the evaluation 
process and understand its value in planning and management will undoubtedly be much more effective than 
programs which view evaluation as an externally imposed requirement which has little value. 

Summary 

The program development process is very challenging. Planning and evaluation in substance abuse 
prevention and treatment is continuously affected by change. It requires extraordinary flexibility, adaptability, 
and creativity to insure that programs are designed to effectively meet the multiple needs of our clientele. We 
must not only be committed to staying abreast of the current best practices in the broad field of human services, 
but we must also critically examine and evaluate those practices in an ongoing quest to find more effective 
approaches to critical problems. This is especially true in the area of substance abuse where treatment outcomes 
have been consistently poor. 

This author has attempted to encourage the reader to not accept the status quo related to program planning 
and evaluation. Instead, it is hoped that the students for whom this book is intended will take up the challenge, 
“to always seek a better way, and as soon as you find it, start looking for an even better way”! To understand the 
process that will enable the reader to respond to this challenge, a six step approach to planning and evaluation 
has been emphasized in this chapter. 

To facilitate the recall of the six steps, the following model is offered to the reader. It is called the RAPPER 
Model. It requires that you use each letter in RAPPER to recall the key first words which describe the steps in 
the process. 
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RAPPER Model 

Review the literature 
Assess the need 

Plan goals/objectives and strategies to achieve them 
Practice those strategies which best address the needs 
Evaluate the effectiveness of the chosen strategies 

Repeat effective strategies. For those ineffective, repeat the process 

Program Development Process 

As a model which emphasizes a commitment to continuous improvement based upon ongoing program 
planning and evaluation, it should be viewed dynamically, like a wheel that is in constant motion. 




It is hoped that the reader does find this model to be a useful tool to recall and follow this process. Mastery 
of this challenging process will enable you to become much more competent in planning, implementing, and 
evaluating programs which specifically address the multiple needs of our target population and offer the best 
chance for success in the prevention and treatment of substance abuse. 
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Program Planning and Evaluation 



Robert E. Rapp 



L Conducting a Comprehensive Needs Assessment 

Rationale 

A comprehensive needs assessment provides the foundation for the program planning process. Knowledge 
about the essential components of comprehensive needs assessments, their importance, how they are developed 
and utilized will enable prospective human services professionals to plan and implement programs that will 
effectively address the multiple needs of their clientele. 

Overview 

This lesson identifies and describes the essential components of a comprehensive needs assessment. It 
emphasizes the importance of each and how they are interrelated. It provides the rationale for using the assessment 
as the foundation for developing program goals, objectives, and methodology. 

Objectives 

• To examine the importance of utilizing social indicators (statistics) to clearly identify the target population 

and to validate their needs for services. 

•To understand the importance of involving the community, including the target population, in the assessment 
process to ensure that the program will be utilized. 

• To become more aware of how agencies often compete for services and/or duplicate services, even 
though significant gaps in services exist, because they fail to collaborate in planning. 

• To understand the importance of including key informants in the assessment so they will have a significant 

role in promoting the need for the program. 

• To understand the importance of offering opportunities for the general public to provide input into the 
assessment and planning process. 



Activities 



Exercise I: Social Indicators 

1 .Divide the students into groups of three or four. Have each group elect a reporter to convey the respective 
group’s ideas to the class as a whole. 

2. Ask each group to determine the best sources to obtain social indicators related to substance abuse. 
Those sources could be national, state, county, city, or even local neighborhoods. 

3. Ask each group to decide whether all sources should be included to present the best picture of need. Ask 
them to determine how social indicators from all sources can be integrated in relation to the others, e.g., 
if you compare the neighborhood’s incidence of substance abuse to the city, county, state, and U.S., will 
the picture of local need become clearer? 

4. Have the groups share their conclusions with the class as a whole. After each group has reported their 
conclusions, summarize and synthesize the information. 

5. Ask the class members to identify situations in which funds or services may be directed to recipients 
who are not members of the target population. 

Exercise II: Community Surveys, Agency Surveys, Open Forums/Meetings and Key Informants. 

1. Divide the students into four groups. Identify a target community or neighborhood in which they will 
conduct a mock needs assessment. 



2. Have group one outline the process of conducting a community survey which will include members of 
the target population. After they have completed that, ask them to develop a rationale to support this 
activity as the most important part of the assessment process. 

3. Have group two outline the process of conducting an agency survey. After they complete that task, ask 
them to develop a rationale to substantiate this activity as the most important part of the assessment 
process. 

4. Have group three outline the process for scheduling and advertising open forums/ meetings to solicit 
input. Have them identify appropriate locations within the given target area. Ask them to develop a 
rationale that substantiates this activity as the most important part of the assessment process. 

5. Have group four outline a process to identify key informants from the given target area. Ask them to 
develop a rationale to substantiate the activity as the most important part of the assessment process. 

6. Have a vocal member of each group debate the importance of their needs assessment activity. 

7. After the debate,have the groups reconvene to carefully consider all the information to determine the 
importance of each activity. Have them rank order the activities in terms of their importance and provide 
a rationale. 

8. Ask each group to report their conclusions. 

II. Formulating Goals and Objectives in Conjunction with a Program Evaluation Plan 

Rationale 

Effective programs have clear goals with measurable specific objectives which are designed to address the 
needs of the target population. As goals and objectives are determined, an evaluation plan should be established 
to document their level of achievement, in order to determine the programs success and to make decisions 
related to improving the program. Successful programs can use their evaluation results to build increased support 
from the community and funding sources. 



Overview 

This lesson differentiates goals from specific objectives. It emphasizes the importance of each and how 
they are interdependent. It provides the rationale to develop an evaluation plan which documents the achievement/ 
lack of achievement of the goals and objectives. 

Objectives 

• To examine the mutuality of goals and objectives, but to promote an understanding that objectives are 

very specific (may identify who, what, when, where) and must be measurable. 

• To be able to easily distinguish goals from objectives. 

• To become more aware that for every goal, there are generally several specific objectives. 

• To understand that some goals and objectives are related to effort (process) while others are related to 
effectiveness (outcome). 

• To be able to easily distinguish effort from effectiveness. 

• To understand that the purpose of program evaluation is to determine the level of 
achievement of goals and objectives by documenting effort, effectiveness, and efficiency. 

• To understand that formulating goals and objectives without developing an evaluation plan invalidates 
the program planning process. 



Activities 

Exercise I: Formulating Goals and Objectives. 

1. Provide an example of a goal, e.g. “Reduce substance abuse among teenagers.” 

2. Provide an example of a specific objective, e.g. “Reduce the rate of DWI among teenagers by 25% 
during the period January I, 1999 and December 31, 1999.” 



3. Ask each student to write down one goal and three specific objectives related to the goal. 

4. Ask students to share their examples. 

5. Emphasize that the first example of a goal is related to effectiveness or outcome, and provide an example 
of a goal related to effort or process; e.g. “All students will receive both group and individual counseling.” 

6. Provide an example of a specific objective related to effort or process, e.g. “One hundred students will 
each receive two hours of individual counseling and two hours of group counseling each month during 
the period January 1, 1999 through December 31, 1999.” 

7. Ask each student to identify whether their goal and objectives are related to effort or effectiveness. Ask 
them to write down one new goal and three specific objectives related to the opposite of their first 
example. 

8. Have each student share both examples and solicit agreement from the other class members that they 
are correctly differentiating between effort and effectiveness. 

Exercise II: Determining an Evaluation Plan, 

1. Assign the class into pairs. 

2. Have each person trade their two sets of goals and objectives. Ask each person to develop a plan to 
collect data that will document the achievement/lack of achievement of his partner’s goals and 
objectives. 

3. Have them share their plans with their partners, discuss them, and if necessary, help the other with an 
evaluation plan. 

4. Have the partners share their examples with the class. 

5. Introduce the concept of efficiency as a concern of program evaluation. Emphasize that it represents 
an acceptable ratio of success between effort and effectiveness as it relates to cost. 

6. Ask students to review the example in the text. Break the class into groups of four or five and ask them 
to think of an example whereby a program’s efficiency would not be acceptable, even though it may 
have achieved its goals and objectives related to effort and effectiveness. 

7. Have each group share their examples with the class and discuss how their example would not satisfy 
the critical question of program efficiency. 

III. Internalizing the Program Development Process 
(Robert Rapp and Jeff Davis) 



Rationale 

The program development process is dynamic. It requires the internalization of an ongoing system of 
planning, practice, and evaluation. Effective systems are flexible, adaptable, and promote creativity in response 
to change. 



Overview 

This lesson is designed to promote better understanding of program development as an ongoing process. It 
is based upon the use of the RAPPER model which emphasizes the interdependence of the six steps of the 
process. 



Objectives 

• To facilitate the recall of the six steps in the process. 

• To understand how each step individually contributes to the process, and how they are all interdependent. 

• To understand that the process is continuous, in that once the planning, practice, and evaluation cycle 
is completed, it is repeated. 

• To understand the purpose of an ongoing system of planning, practice and evaluation is tied to 
a constant and a quest for quality. 
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Activities 



1. Divide the students into six groups, one group for each step in the RAPPER model. Instruct each 
group to elect a reporter to convey the respective group’s ideas to the class. 

2. Hand out a sample written scenario to each group. The scenario should include all of the objective 
parameters of a human services program. An example could be: 

A local community mental health agency has recently acquired a contract for a high school 
prevention program to address the substance abuse of its students. The school’s records from 
the previous year indicate that 25% of its student population is actively abusing substances of 
some kind. The agency has 25 hours-per-week for it’s program. 

• Scenario should be modified or created to fit the demographics of the students’ community. 

3. Randomly assign each group one of the six steps of the RAPPER model of the Program Development 
Process; i.e., one group would have “Assess the need,” another “Evaluate the effectiveness. . .”, etc. 

4. Instruct each group to brainstorm strategies, information, or resources that they might need to implement 
their step as it relates to the scenario. Emphasize to the individual groups the importance of not 
censoring their ideas at this point, but to allow consideration of all ideas from each student. Emphasize 
further that there is to be no exchange of ideas between groups at this point. 

5. Instruct each group to come to a consensus as to which seem to be the most useful ideas/information 
sources/plans for their group’s step, and to record them on one sheet of paper. 

6. Come together as a class. Instruct each reporter to present the group’s step to the class. 

7. Lead a process discussion after the presentations. Topics for discussion may include: 

• Ways in which many of the steps are similar/ways in which they differ. 

• Interdependence of steps, i.e. sources of information that are useful or may be useful for more than 
one step. 

• Topics that arise that are unique to the particular area and do not necessarily generalize to other 
areas with this particular program. 

• How many different useful perspectives can come from one scenario. 
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